feat(core): update document in vector store #210

mackurzawa · 2024-11-28T08:28:18Z

Description

This PR ensures that multiple ingestions of the same documents (in terms of source.id) do not retain VectorStoreEntries related to previous versions of the documents in the database. Before each ingestion, the source.id of the documents to be ingested is checked, and all VectorStoreEntries with the same source.id are removed from the database. The ingestion then proceeds as usual.

Implemented logic to check if the document's id is already present in VectorStore.

Implemented the remove() method for ChromaVectorStore, QdrantVectorStore, and InMemoryVectorStore.

Implemented 3 unit tests for each VectorStore to verify the functionality of the remove() methods.

Implemented 3 integration tests for each VectorStore with the following steps:

Create two text documents, A and B, with content.
Ingest documents A and B.
Change the content of document B.
Ingest document B.
Verify that the database contains the content of document A.
Verify that the database contains the new content of document B.
Verify that the database does not contain the old content of document B.

Testing

Testing can be done by running any of the integration tests:
packages/ragbits-core/tests/integration/vector_stores/test_chroma.py
packages/ragbits-core/tests/integration/vector_stores/test_in_memory.py
packages/ragbits-core/tests/integration/vector_stores/test_qdrant.py
or unit tests named in the below files:
packages/ragbits-core/tests/unit/vector_stores/test_chroma.py
packages/ragbits-core/tests/unit/vector_stores/test_in_memory.py
packages/ragbits-core/tests/unit/vector_stores/test_qdrant.py

…empty store

… with same source.id

…cuments with the same source

github-actions · 2024-11-28T08:32:27Z

Trivy scanning results.

.venv/lib/python3.10/site-packages/PyJWT-2.9.0.dist-info/METADATA (secrets)

Total: 1 (MEDIUM: 1, HIGH: 0, CRITICAL: 0)

MEDIUM: JWT (jwt-token)
════════════════════════════════════════
JWT token
────────────────────────────────────────
.venv/lib/python3.10/site-packages/PyJWT-2.9.0.dist-info/METADATA:80
────────────────────────────────────────
78 >>> encoded = jwt.encode({"some": "payload"}, "secret", algorithm="HS256")
79 >>> print(encoded)
80 [ *********************************************************************************************************
81 >>> jwt.decode(encoded, "secret", algorithms=["HS256"])
────────────────────────────────────────

.venv/lib/python3.10/site-packages/litellm/llms/huggingface_llms_metadata/hf_text_generation_models.txt (secrets)

Total: 1 (MEDIUM: 0, HIGH: 0, CRITICAL: 1)

CRITICAL: HuggingFace (hugging-face-access-token)
════════════════════════════════════════
Hugging Face Access Token
────────────────────────────────────────
.venv/lib/python3.10/site-packages/litellm/llms/huggingface_llms_metadata/hf_text_generation_models.txt:36162
────────────────────────────────────────
36160 mncai/Llama2-7B-Active_3rd-floor-LoRA-dim64_epoch4
36161 ajcdp/CM
36162 [ Nagharjun17/*************************************
36163 BigSalmon/InformalToFormalLincoln114Paraphrase
────────────────────────────────────────

.venv/lib/python3.10/site-packages/litellm/proxy/_types.py (secrets)

Total: 1 (MEDIUM: 1, HIGH: 0, CRITICAL: 0)

MEDIUM: Slack (slack-web-hook)
════════════════════════════════════════
Slack Webhook
────────────────────────────────────────
.venv/lib/python3.10/site-packages/litellm/proxy/_types.py:1288
────────────────────────────────────────
1286 alert_to_webhook_url: Optional[Dict] = Field(
1287 None,
1288 [ bhook_url: {'budget_alerts': '*****************************************************************************'}`",
1289 )
────────────────────────────────────────

…rant test_list

packages/ragbits-core/src/ragbits/core/vector_stores/qdrant.py

packages/ragbits-document-search/src/ragbits/document_search/_main.py

github-actions · 2024-11-28T12:27:02Z

Code Coverage Summary

Filename                                                                                                      Stmts    Miss  Cover    Missing
----------------------------------------------------------------------------------------------------------  -------  ------  -------  -------------------------------------
packages/__init__.py                                                                                              0       0  100.00%
packages/ragbits-core/src/ragbits/core/__init__.py                                                                0       0  100.00%
packages/ragbits-core/src/ragbits/core/config.py                                                                  7       0  100.00%
packages/ragbits-core/src/ragbits/core/audit/__init__.py                                                         67       6  91.04%   40-48
packages/ragbits-core/src/ragbits/core/audit/base.py                                                             32       0  100.00%
packages/ragbits-core/src/ragbits/core/audit/otel.py                                                             36      16  55.56%   20-21, 35-41, 51-54, 64-67, 96
packages/ragbits-core/src/ragbits/core/embeddings/__init__.py                                                    11       0  100.00%
packages/ragbits-core/src/ragbits/core/embeddings/base.py                                                        12       2  83.33%   44, 56
packages/ragbits-core/src/ragbits/core/embeddings/noop.py                                                         6       0  100.00%
packages/ragbits-core/src/ragbits/core/llms/__init__.py                                                          14       8  42.86%   27-39
packages/ragbits-core/src/ragbits/core/llms/base.py                                                              47       9  80.85%   46, 65, 149-157, 160-162
packages/ragbits-core/src/ragbits/core/llms/factory.py                                                           18       3  83.33%   48, 59, 69
packages/ragbits-core/src/ragbits/core/llms/litellm.py                                                           40      13  67.50%   9-10, 55, 86, 92-113
packages/ragbits-core/src/ragbits/core/llms/types.py                                                              9       2  77.78%   25, 29
packages/ragbits-core/src/ragbits/core/llms/clients/__init__.py                                                   4       0  100.00%
packages/ragbits-core/src/ragbits/core/llms/clients/base.py                                                      26       0  100.00%
packages/ragbits-core/src/ragbits/core/llms/clients/exceptions.py                                                17       7  58.82%   7-8, 17, 26-27, 36, 45
packages/ragbits-core/src/ragbits/core/llms/clients/litellm.py                                                   77      21  72.73%   11-12, 72, 120, 155-176, 196-201, 212
packages/ragbits-core/src/ragbits/core/llms/clients/local.py                                                     52      24  53.85%   10-13, 65-73, 94-105, 126-142
packages/ragbits-core/src/ragbits/core/metadata_stores/__init__.py                                               12       3  75.00%   27-30
packages/ragbits-core/src/ragbits/core/metadata_stores/base.py                                                    6       0  100.00%
packages/ragbits-core/src/ragbits/core/metadata_stores/exceptions.py                                              4       0  100.00%
packages/ragbits-core/src/ragbits/core/metadata_stores/in_memory.py                                              16       0  100.00%
packages/ragbits-core/src/ragbits/core/prompt/__init__.py                                                         2       0  100.00%
packages/ragbits-core/src/ragbits/core/prompt/base.py                                                            20       0  100.00%
packages/ragbits-core/src/ragbits/core/prompt/parsers.py                                                         35       0  100.00%
packages/ragbits-core/src/ragbits/core/prompt/prompt.py                                                         126       0  100.00%
packages/ragbits-core/src/ragbits/core/prompt/discovery/__init__.py                                               2       0  100.00%
packages/ragbits-core/src/ragbits/core/prompt/discovery/prompt_discovery.py                                      33       2  93.94%   54-55
packages/ragbits-core/src/ragbits/core/utils/__init__.py                                                          0       0  100.00%
packages/ragbits-core/src/ragbits/core/utils/_pyproject.py                                                       37       0  100.00%
packages/ragbits-core/src/ragbits/core/utils/config_handling.py                                                  16       4  75.00%   32-33, 37-38
packages/ragbits-core/src/ragbits/core/utils/decorators.py                                                       29       0  100.00%
packages/ragbits-core/src/ragbits/core/utils/dict_transformations.py                                             72       3  95.83%   24, 27, 108
packages/ragbits-core/src/ragbits/core/vector_stores/__init__.py                                                  8       0  100.00%
packages/ragbits-core/src/ragbits/core/vector_stores/base.py                                                     28       1  96.43%   65
packages/ragbits-core/src/ragbits/core/vector_stores/chroma.py                                                   56       3  94.64%   57-58, 75
packages/ragbits-core/src/ragbits/core/vector_stores/in_memory.py                                                36       0  100.00%
packages/ragbits-core/src/ragbits/core/vector_stores/qdrant.py                                                   59       3  94.92%   54-55, 75
packages/ragbits-core/tests/integration/vector_stores/test_vector_store.py                                       31       0  100.00%
packages/ragbits-core/tests/unit/__init__.py                                                                      0       0  100.00%
packages/ragbits-core/tests/unit/audit/__init__.py                                                                0       0  100.00%
packages/ragbits-core/tests/unit/audit/test_otel.py                                                               7       0  100.00%
packages/ragbits-core/tests/unit/audit/test_trace.py                                                             88       3  96.59%   13, 16, 19
packages/ragbits-core/tests/unit/llms/__init__.py                                                                 0       0  100.00%
packages/ragbits-core/tests/unit/llms/test_litellm.py                                                            64       0  100.00%
packages/ragbits-core/tests/unit/llms/factory/__init__.py                                                         3       0  100.00%
packages/ragbits-core/tests/unit/llms/factory/test_get_default_llm.py                                            10       0  100.00%
packages/ragbits-core/tests/unit/llms/factory/test_get_llm_from_factory.py                                        8       0  100.00%
packages/ragbits-core/tests/unit/metadata_stores/__init__.py                                                      0       0  100.00%
packages/ragbits-core/tests/unit/metadata_stores/test_in_memory.py                                               22       0  100.00%
packages/ragbits-core/tests/unit/prompts/__init__.py                                                              0       0  100.00%
packages/ragbits-core/tests/unit/prompts/test_parsers.py                                                         65       0  100.00%
packages/ragbits-core/tests/unit/prompts/test_prompt.py                                                         165       0  100.00%
packages/ragbits-core/tests/unit/prompts/discovery/__init__.py                                                    0       0  100.00%
packages/ragbits-core/tests/unit/prompts/discovery/prompt_classes_for_tests.py                                   30       0  100.00%
packages/ragbits-core/tests/unit/prompts/discovery/test_prompt_discovery.py                                      18       0  100.00%
packages/ragbits-core/tests/unit/prompts/discovery/ragbits_tests_pkg_with_prompts/__init__.py                     2       1  50.00%   3
packages/ragbits-core/tests/unit/prompts/discovery/ragbits_tests_pkg_with_prompts/prompts/__init__.py             3       2  33.33%   2-4
packages/ragbits-core/tests/unit/prompts/discovery/ragbits_tests_pkg_with_prompts/prompts/temp_prompt1.py        14       0  100.00%
packages/ragbits-core/tests/unit/prompts/discovery/ragbits_tests_pkg_with_prompts/prompts/temp_prompt2.py        14       0  100.00%
packages/ragbits-core/tests/unit/utils/__init__.py                                                                0       0  100.00%
packages/ragbits-core/tests/unit/utils/test_decorators.py                                                        26       2  92.31%   17, 39
packages/ragbits-core/tests/unit/utils/test_dict_transformations.py                                              69       0  100.00%
packages/ragbits-core/tests/unit/utils/pyproject/test_find.py                                                    13       0  100.00%
packages/ragbits-core/tests/unit/utils/pyproject/test_get_config.py                                               9       0  100.00%
packages/ragbits-core/tests/unit/utils/pyproject/test_get_instace.py                                             37       0  100.00%
packages/ragbits-core/tests/unit/vector_stores/__init__.py                                                        0       0  100.00%
packages/ragbits-core/tests/unit/vector_stores/test_chroma.py                                                    61       0  100.00%
packages/ragbits-core/tests/unit/vector_stores/test_in_memory.py                                                 84       0  100.00%
packages/ragbits-core/tests/unit/vector_stores/test_qdrant.py                                                    40       0  100.00%
packages/ragbits-document-search/src/ragbits/document_search/__init__.py                                          2       0  100.00%
packages/ragbits-document-search/src/ragbits/document_search/_main.py                                            86       4  95.35%   187-194, 197
packages/ragbits-document-search/src/ragbits/document_search/documents/__init__.py                                0       0  100.00%
packages/ragbits-document-search/src/ragbits/document_search/documents/document.py                               61       2  96.72%   99, 146
packages/ragbits-document-search/src/ragbits/document_search/documents/element.py                                78      12  84.62%   87, 167-174, 183-185
packages/ragbits-document-search/src/ragbits/document_search/documents/exceptions.py                             11       5  54.55%   7-8, 17, 26-27
packages/ragbits-document-search/src/ragbits/document_search/documents/sources.py                               116      13  88.79%   130, 213-218, 255-258, 262-263
packages/ragbits-document-search/src/ragbits/document_search/ingestion/__init__.py                                0       0  100.00%
packages/ragbits-document-search/src/ragbits/document_search/ingestion/document_processor.py                     28       0  100.00%
packages/ragbits-document-search/src/ragbits/document_search/ingestion/processor_strategies/__init__.py          12       1  91.67%   31
packages/ragbits-document-search/src/ragbits/document_search/ingestion/processor_strategies/base.py              25       0  100.00%
packages/ragbits-document-search/src/ragbits/document_search/ingestion/processor_strategies/batched.py           18       0  100.00%
packages/ragbits-document-search/src/ragbits/document_search/ingestion/processor_strategies/distributed.py       30       8  73.33%   8-9, 36, 64-71
packages/ragbits-document-search/src/ragbits/document_search/ingestion/processor_strategies/sequential.py        13       0  100.00%
packages/ragbits-document-search/src/ragbits/document_search/ingestion/providers/__init__.py                     10       0  100.00%
packages/ragbits-document-search/src/ragbits/document_search/ingestion/providers/base.py                         14       0  100.00%
packages/ragbits-document-search/src/ragbits/document_search/ingestion/providers/dummy.py                        20       7  65.00%   33, 54-60
packages/ragbits-document-search/src/ragbits/document_search/ingestion/providers/unstructured/__init__.py         4       0  100.00%
packages/ragbits-document-search/src/ragbits/document_search/ingestion/providers/unstructured/default.py         46       4  91.30%   98, 103-104, 137
packages/ragbits-document-search/src/ragbits/document_search/ingestion/providers/unstructured/images.py          50      19  62.00%   73-80, 87-99, 111, 124
packages/ragbits-document-search/src/ragbits/document_search/ingestion/providers/unstructured/pdf.py             19       6  68.42%   23, 35-43
packages/ragbits-document-search/src/ragbits/document_search/ingestion/providers/unstructured/utils.py           38      11  71.05%   71, 82-83, 98-101, 110, 121-123
packages/ragbits-document-search/src/ragbits/document_search/retrieval/__init__.py                                0       0  100.00%
packages/ragbits-document-search/src/ragbits/document_search/retrieval/rephrasers/__init__.py                    15       4  73.33%   39-44
packages/ragbits-document-search/src/ragbits/document_search/retrieval/rephrasers/base.py                         7       1  85.71%   32
packages/ragbits-document-search/src/ragbits/document_search/retrieval/rephrasers/llm.py                         22       9  59.09%   28-29, 47-50, 67-69
packages/ragbits-document-search/src/ragbits/document_search/retrieval/rephrasers/noop.py                         6       0  100.00%
packages/ragbits-document-search/src/ragbits/document_search/retrieval/rephrasers/prompts.py                     16       4  75.00%   49-54
packages/ragbits-document-search/src/ragbits/document_search/retrieval/rerankers/__init__.py                     10       1  90.00%   27
packages/ragbits-document-search/src/ragbits/document_search/retrieval/rerankers/base.py                         15       0  100.00%
packages/ragbits-document-search/src/ragbits/document_search/retrieval/rerankers/litellm.py                      18       0  100.00%
packages/ragbits-document-search/src/ragbits/document_search/retrieval/rerankers/noop.py                         11       0  100.00%
packages/ragbits-document-search/tests/__init__.py                                                                0       0  100.00%
packages/ragbits-document-search/tests/helpers.py                                                                 3       0  100.00%
packages/ragbits-document-search/tests/integration/__init__.py                                                    0       0  100.00%
packages/ragbits-document-search/tests/integration/test_rerankers.py                                             15       6  60.00%   18-38
packages/ragbits-document-search/tests/integration/test_sources.py                                               23      10  56.52%   22-32, 40-45
packages/ragbits-document-search/tests/integration/test_unstructured.py                                          47      10  78.72%   46-52, 65-71
packages/ragbits-document-search/tests/unit/test_document_processor.py                                           17       0  100.00%
packages/ragbits-document-search/tests/unit/test_document_search.py                                              87       0  100.00%
packages/ragbits-document-search/tests/unit/test_documents.py                                                    13       0  100.00%
packages/ragbits-document-search/tests/unit/test_elements.py                                                     19       0  100.00%
packages/ragbits-document-search/tests/unit/test_local_file_source.py                                            13       0  100.00%
packages/ragbits-document-search/tests/unit/test_processing_strategies.py                                        25       0  100.00%
packages/ragbits-document-search/tests/unit/test_providers.py                                                    31       0  100.00%
packages/ragbits-document-search/tests/unit/test_rerankers.py                                                    31       1  96.77%   21
packages/ragbits-document-search/tests/unit/test_source_discriminator.py                                         35       0  100.00%
packages/ragbits-document-search/tests/unit/test_sources.py                                                      25       0  100.00%
packages/ragbits-guardrails/src/ragbits/guardrails/__init__.py                                                    0       0  100.00%
packages/ragbits-guardrails/src/ragbits/guardrails/base.py                                                       15       0  100.00%
packages/ragbits-guardrails/src/ragbits/guardrails/openai_moderation.py                                          19       5  73.68%   29-33
packages/ragbits-guardrails/tests/unit/test_openai_moderation.py                                                 35       0  100.00%
TOTAL                                                                                                          3209     281  91.24%

Diff against main

Filename                                                                             Stmts    Miss  Cover
---------------------------------------------------------------------------------  -------  ------  --------
packages/ragbits-core/src/ragbits/core/vector_stores/base.py                            +2       0  +0.28%
packages/ragbits-core/src/ragbits/core/vector_stores/chroma.py                          +3       0  +0.30%
packages/ragbits-core/src/ragbits/core/vector_stores/in_memory.py                       +4       0  +100.00%
packages/ragbits-core/src/ragbits/core/vector_stores/qdrant.py                          +8       0  +0.80%
packages/ragbits-core/tests/integration/vector_stores/test_vector_store.py             +31       0  +100.00%
packages/ragbits-core/tests/unit/vector_stores/test_chroma.py                           +5       0  +100.00%
packages/ragbits-core/tests/unit/vector_stores/test_in_memory.py                        +6       0  +100.00%
packages/ragbits-core/tests/unit/vector_stores/test_qdrant.py                           +7       0  +100.00%
packages/ragbits-document-search/src/ragbits/document_search/_main.py                   +9       0  +0.54%
packages/ragbits-document-search/src/ragbits/document_search/documents/element.py       +1       0  +0.20%
TOTAL                                                                                  +76       0  +0.21%

Results for commit: e84f110

Minimum allowed coverage is 60%

♻️ This comment has been updated with latest results

…ries_with_same_sources

…iency

…dling_document_ingestion_with_different_content_and_verifying_replacement

packages/ragbits-document-search/src/ragbits/document_search/_main.py

packages/ragbits-core/src/ragbits/core/vector_stores/qdrant.py

…ve_entries_with_same_sources

packages/ragbits-core/tests/integration/vector_stores/test_chroma.py

packages/ragbits-core/tests/unit/vector_stores/test_chroma.py

packages/ragbits-document-search/src/ragbits/document_search/_main.py

mhordynski

lgtm

mackurzawa added 7 commits November 27, 2024 17:31

fix: qdrant returns empty list instead of raising error when listing …

22d775d

…empty store

fix: add document_meta.source.id to VectorStoreEntry metadata

885a77d

feat: add remove methods for vector stores

f07b293

test: add unit tests for remove method in vector stores

0080c8f

feat: add removing entries from vector store when ingesting documents…

2857180

… with same source.id

test: add integration tests for vector stores, regarding ingesting do…

44f3ea7

…cuments with the same source

fix: lint

42c6afd

mackurzawa linked an issue Nov 28, 2024 that may be closed by this pull request

feat: updating document in a vector store #194

Closed

mackurzawa changed the title ~~Feat/update document in vector store~~ feat(core): update document in vector store Nov 28, 2024

fix: change checking qdrant collection existance method and update qd…

af80819

…rant test_list

ludwiktrammer reviewed Nov 28, 2024

View reviewed changes

packages/ragbits-core/src/ragbits/core/vector_stores/qdrant.py Show resolved Hide resolved

packages/ragbits-document-search/src/ragbits/document_search/_main.py Outdated Show resolved Hide resolved

fix: mock embedder in integration tests

c8870ab

mackurzawa added 3 commits November 28, 2024 13:50

chore: rename function from remove_entries_with_same_id to remove_ent…

c042a33

…ries_with_same_sources

refactor: optimize integration tests for better readability and effic…

cb2d051

…iency

chore: rename integration tests from test_update_document to test_han…

feccfc6

…dling_document_ingestion_with_different_content_and_verifying_replacement

mackurzawa marked this pull request as ready for review November 28, 2024 15:19

Merge branch 'main' into feat/update-document-in-vector-store

eec2bfc

ludwiktrammer approved these changes Nov 28, 2024

View reviewed changes

mhordynski requested changes Nov 28, 2024

View reviewed changes

mackurzawa added 2 commits November 29, 2024 08:24

chore: make remove_entries_with_same_sources private

1e97250

merge main and resolve conflicts

7eea03e

micpst reviewed Nov 29, 2024

View reviewed changes

packages/ragbits-core/src/ragbits/core/vector_stores/qdrant.py Show resolved Hide resolved

mackurzawa added 3 commits November 29, 2024 12:30

fix: handle limit=None in Qdrant

96e9948

chore: add TODO for passing 'where' argument to list method in __remo…

b48a1a2

…ve_entries_with_same_sources

fix: change way for obtaining metadata from entry

3d8a32f

pawel-chmielak-deepsense reviewed Nov 29, 2024

View reviewed changes

mackurzawa and others added 2 commits November 29, 2024 14:06

docs: update docstring for qdrant remove method

887f946

Merge branch 'main' into feat/update-document-in-vector-store

2ad8a4a

mackurzawa added 2 commits November 29, 2024 15:27

chore: update function name

a09bf2c

refactor: combine integration tests to one parametrized test

e84f110

mhordynski approved these changes Nov 29, 2024

View reviewed changes

mackurzawa merged commit 2816e60 into main Nov 29, 2024
6 checks passed

mackurzawa deleted the feat/update-document-in-vector-store branch November 29, 2024 14:52

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(core): update document in vector store #210

feat(core): update document in vector store #210

mackurzawa commented Nov 28, 2024

github-actions bot commented Nov 28, 2024 •

edited

Loading

github-actions bot commented Nov 28, 2024 •

edited

Loading

mhordynski left a comment

feat(core): update document in vector store #210

feat(core): update document in vector store #210

Conversation

mackurzawa commented Nov 28, 2024

Description

Testing

github-actions bot commented Nov 28, 2024 • edited Loading

.venv/lib/python3.10/site-packages/PyJWT-2.9.0.dist-info/METADATA (secrets)

.venv/lib/python3.10/site-packages/litellm/llms/huggingface_llms_metadata/hf_text_generation_models.txt (secrets)

.venv/lib/python3.10/site-packages/litellm/proxy/_types.py (secrets)

github-actions bot commented Nov 28, 2024 • edited Loading

Code Coverage Summary

Diff against main

mhordynski left a comment

Choose a reason for hiding this comment

github-actions bot commented Nov 28, 2024 •

edited

Loading

github-actions bot commented Nov 28, 2024 •

edited

Loading